AD  683750 


4JFOSR  69*  0445TK 

1 


Tests  o£  indexes.  A  comparison  of  keyword 
from  title  indexes  with  and  without  added 
keywords  and  a  single  access  point  per  docu¬ 
ment  alphabetic  subject  index* 

*V  *  •«’ 


Report  by: 

G.  Jahoda 
Mary  Lou  Stursa 

Work  by: 

Mary  Culnan 
G.  Jahoda 
John  Lattimer 
Mary  Lou  Stursa 


January  1969 


School  of  Library  Science 
Florida  State  University 
Tallahassee,  Florida 


D  D  C 


*Work  supported  in  part  by  the  Air  Fnrce  Office  of  Scientific 
Research,  Office  of  Aerospace  Research,  United  States  Air 
Force,  under  AFOSR  grant  number  895-65. 


1*  This  document  has  been  approved  for  publlo 
release  and  sale  ;  its  distribution  is  unlimited. 


Best  Available  Copy 


Reproduced  by  Ibe 

CLEARINGHOUSE 
lor  Federal  Scientific  &  Technical 
Information  Springfield  Vs.  22151 


Table  of  Contents 


Page 

Table  of  Contents . . . - . .  II 

List  of  Figures . . .  Ill 

List  of  Tables . . .  1^ 

Abstract . ...... . . .  1 

I.  Introduction . . .  1 

II.  Description  of  work . 2 

a.  The  size  of  the  document  collection  and  the  variables 

tested . . . 2 

b.  The  questions  searched  and  the  search  procedure . 5 

c.  Measures  of  search  results.... .  8 

III.  Search  results .  12 

a.  Group  I  and  Group  71;  Index  A  and  Ind<_;c  B;  Index  A  and 

Index  E . . . 12 

b .  Index  C  and  Index  D . 32 

IV.  Discussion  of  search  results . . . . . .  34 

a.  Index  A  and  Index  B?  Index  A  and  Index  E . 34 

d.  Single  figure  search  measures . . .  39 

c.  Index  C  and  Index  D . 41 

d.  Some  general  conclusions  and  suggestions  for  future  work...  42 

V  .  Bibliography  .  . . 45 

VI .  Acknowledgments . 46 

Appendixes : 

A.  List  of  questions  used  in  tests .  47 

B.  Instructions  to  searchers . . . . . 51 

1 1 


List  of  Figures 


? 


t 


j  * 


i 


Figure  Page 

1  Characterization  of  Indexes  used  in  Tests . . . 3 

2  Number  of  Relevant  Documents  per  Question . . 6 

3  Recall  -  Group  I  and  Group  II .  13 

4  Recall  -  Index  A  and  Index  B . . . 14 

5  Recall  -  Index  A  and  Index  E . 15 

6  Precision  -  Group  I  and  Group  II . . . .  ....  16 

7  Tndex  A  and  Index  B  -  Precision......... . 17 

8  Precision  -  Index  A  and  Index  E . 18 

9  Time  -  Group  I  and  Group  II . .19 

10  Time  -  Index  A  and  Index  B . 20 

11  Time  -  Index  A  and  Index  E  . . 21 

12  TRF  -  Group  I  and  Group  II . . . 22 

13  TRF  -  Index  A  and  Index  B . 23 

14  TRF  -  Index  A  and  Index  E . . . . . .  .  24 

15  ERF  -  Group  I  and  Group  II . 25 

16  ERF  -  Index  A  and  Index  B „ . 26 

17  ERF  -  Index  A  and  Index  £.., . 27 

18  E  -  Group  I  and  Group  II . 28 

19  E  -  Index  A  and  Index  B . . . 29 

20  E  -  Index  A  and  Index  E . 30 

21  Number  of  Documents  per  Folder . . .  38 


t 


III 


List  of  Tables 


Table  Page 

1  Comparison  of  Search  Results  of  Group  I 

searchers  and  excluded  searchers . . . .  7 

2  Search  Pattern  for  Group  I  and  Group  II . .  9 


3  Summary  of  Search  Results,  Groups  I  and  II; 
Indexes  A  and  B;  Indexes  A  and  E . 


4  Results  of  Tests  of  Statistical  Difference .  33 

5  Index  A  -  delete  words  that  might  have  been  used  to 

retrieve  relevant  documents . . .  34 

6  Cross-references  for  Index  D  that  might  have  been 

used  to  retrieve  relevant  documents . . . .  35 


1 


X 


Abstract 

Three  indexes  to  a  collection  of  3204  documents  in  the  field  of 
i.v»emistry  were  test  searched.  The  indexes  are  a  keyword  from  title  index 
without  added  keywords,  a  keyword  from  title  index  with  added  keywords, 
and  a  single  access  pc^nt  per  document  alphabetic  subject  index.  The 
three  indexes  were  searched  by  14  graduate  chemistry  students  with  66 
questions.  Search  results  are  characterized  in  terms  of  recall,  pre¬ 
cision,  search  time,  and  three  other  single  figure  measures.  A  measure 
of  index  performance  bared  on  recall,  precision  and  search  time  was 
developed.  There  is  no  statistically  significant  difference  between 
search  results  with  the  muluiplt  access  points  per  document  keyword  from 
title  index  and  the  single  access  point  per  document  alphabetic  subject 
index.  Statistically  significant  differences  in  search  results  were 
found  between  the  keyword  from  title  index  with  added  keywords  and  the 
keyword  from  title  index  without  added  keywords.  The  effect  of  the  size 
of  the  delete  word  list  and  of  elements  of  vocabulary  control  in  keyword 
from  title  indexes  were  also  studied. 


I.  Introduction 

Several  hundred  records  of  researchers '  document  use  have  beer  _ol- 
lected  as  part  of  a  study  of  information  gathering  habits.  Thirteen 
researchers  in  science  and  engineering  at  Florida  State  University  par¬ 
ticipated  in  this  study  of  use  of  documents  in  the  researchers'  personal 
document  collections.  Each  record  of  use  consists  of  a  record  of  the 
queotion  and  a  record  of  the  bibliographic  citation  cf  documents  that  the 
reeearcher  considered  relevant  for  the  question.  The  case  histories  of 
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document  use  were  collected  to  gain  additional  insight  into  information 
gathering  habits  of  researchers  and  also  to  determine  whether  information 
specialists  can  be  of  assistance  in  the  preparation  of  indexes  to  re¬ 
searchers'  personal  document  collections.  For  further  background  on 
this  aspect  of  the  study,  the  reader  is  referred  to  a  publication  by 
Jahoda,  Hutchins,  and  Miller.1-  Gix  keyword  from  title  indexes  have  been 
prepared  for  the  personal  document  collections  of  six  researchers. 

Three  of  these  indexes,  the  indexes  to  the  document  collections  of  three 
chemists,  are  used  in  the  tests  of  indexes  that  are  described  in  this 
report.  These  tests  were  intended  to  answer  the  following  questions  for 
indexes  to  about  3200  documents : 

*  What  is  the  effect  of  index  depth  on  index  performance? 

*  What  is  the  effect  of  added  keywords,  size  of  delete  list,  and 

elements  of  vocabulary  control  on  the  performance  of  keyword  from  title 
indexes? 

II.  Description  of  work 

a.  The  size  of  the  document  collection  and  the  index  variables  tested 
The  derivation  of  the  five  indexes  used  in  the  test  and  their 

characteristics  are  outlined  in  Fig.  1.  Each  of  the  five  indexes  is  to 

the  same  collection  of  3204  documents.  These  3204  documents  represent  the 
merged  document  collections  from  the  offices  of  three  chemists  for  whom 
keyword  out  of  context  (KWOC)  indexes  have  been  prepared.  Two  of  the 
three  chemists  added  keywords  to  titles  in  their  KWOC  indexes;  the  third 
chemist  did  not.  Each  of  the  chemists  selected  his  own  list  of  delete 
words.  The  three  individual  lists  were  merged  into  one  list  of  628  delete 
words  which  was  used  in  the  preparation  of  the  merged  KWOC  index.  The 
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Chemist  1 
2039  documents 
KWOC  titles 
only 


Chemist  2 
467  documents 
KWOC  titles  + 
added  keywords 


Chemist  3 
698  documents 
KWOC  titles  + 
added  keywords 


Merged  index  of 
3204  documents 


Ind*>v 

3204 

ments 

delet 

words 

words 

title 


Index 


docu- 
;  628 


3204  docu¬ 
ments;  628 
delete 
words ; 
words  in 
title  t 
added  key¬ 
words  from 
chemists  2 
and  3 


Index  C 
3294  docu 
ments;  16 
delete 
words ; 
words  i n 
t '  *■  le 


Index  P 
3204  docu¬ 
ments;  629 
delete 
words ; 
words  in 
title 

only;  ele¬ 
ments  ot 
vocabulary 
control 


index  E 
3204  docu¬ 
ments  ; 
single  ac¬ 
cess  point/ 
document ; 
subject 
filing  in 
subject 
folders 


Fig.  1 , 


Characterization  of  indexes  used  m  tests 
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merging  of  the  three  chemists'  KWuC  indexes  ! two  of  which  had  additional 
keywords)  yielded  Index  B.  Index  A,  the  index  that  is  used  as  the  basis 
of  comparison  against  the  other  four  indexes,  was  prepared  by  removing 
the  additional  keywords  from  Index  B.  Index  A  is  thus  the  same  as 
Index  B  except  that  it  does  not  have  the  added  keywords. 
i  Index  C  was  to  have  been  the  same  as  Index  A  but  it  was  to  have  a 

minimum  delete  word  list  of  16  words  instead  of  the  list  of  628  delete 
words  in  Indexes  A,  B  and  D.  The  list  of  delete  words  in  Index  C  are 
the  j.oll'-ving  16  words  which  have  been  used  by  H.  P.  Luhn  in  the  prepara¬ 
tion  of  one  of  his  keyword  from  title  indexes: 

a,  an,  and,  as, at,  by,  for,  from,  if,  in,  of,  on,  or,  the,  to, 
with . ? 

The  size  of  the  delete  word  list  is  therefore  the  variable  to  be  tested 
in  Index  C. 

Index  D  differs  from  Index  A  in  that  it  was  tc  contain  the  following 

? 

elements  of  vocabulary  control,  the  variable  to  be  tested  in  this  index: 

*  lumping  of  singular  and  plural  forms  cf  nouns 

*  lumping  of  words  that  differ  from  each  other  only  in  word  form, 
e.  g . ,  analysis,  analyzing,  analysed 

*  "fee"  cross-references  t or  synonyms 

*  "fee  also"  cross-references  for  word  relationships  other  than 
spec  1 f i c-g^ner 1 c  relationships ,  e.  c.,  ion- exchange  and  cation 
ex chance . 

Kach  cf  the  1204  documents  included  in  the  KWOC  index  is  filed  in 
a  subject  folder  in  one  of  the  three  chemists’  offices.  The  list  of 
these  subject  folder  headings  ra.  be  considered  as  the  index  headings  cf 
an  alphabetic  subject  index.  Index  H  is  ■  sincle  access  reir.t  per  document 
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alphabetic  subject  index.  It  consists  of  an  alphabet  listing  of  the 
subject  headings  under  which  the  documents  are  filed  along  with  the 
bibliographic  citation  (including  title)  of  eac<.  ^ocunent  so  filed. 

b.  The  questions  searched  and  the  search  procedure 

The  indexes  were  searched  with  66  questions.  The  questions 
were  selected  from  the  Cc.se  histories  of  use  of  the  documents  in  the  three 
chemists'  offices.  Not  all  of  the  three  chemists'  recorded  case  histories 
were  used  in  the  test.  Questions  that  were  for  the  recall  of  a  specific 
document,  questions  that  were  almost  identical  to  questions  used,  and 
questions  that  were  primarily  for  general  background  rather  than  for 
everything  in  the  files  on  a  subject  were  all  omitted  from  the  test.  The 
case  histories  (the  question  and  the  document  or  documents  considered 
relevant  by  the  chemist  searching  his  own  collection)  were  collected 
before  the  individual  KWOC  indexes  had  been  prepared.  The  list  of  rele¬ 
vant.  documents  for  each  question  was  reviewed  by  the  chemises  before  the 
tests  were  started.  During  this  review  the  chemists  selected  additional 
relevant  documents  that  were  either  not  in  the  collection  or  were  missed 
when  the  original  searches  were  made.  The  6b  questions  used  in  the  test 
are  listed  in  Appendix  A.  Flu.  2  gives  a  tabulation  of  the  number  of 
relevant  documents  per  question. 

Three  of  the  indexes.  Indexes  A,  B  and  E,  were  searched  by  14  q.aduate 
chemistry  students.  The  searchers  were  given  both  written  and  oral  in¬ 
structions  on  the  use  of  the  indexes  before  they  started  the  test  seal ches . 
The  written  instructions  are  reproducer,  as  Appendix  B.  Each  searcher 
also  conducted  a  trial  search  under  the  supervision  cf  a  member  of  the 
project  staff.  Search  results  in  the  trial  search  -ere  r.ct  counted  m 


the  overall  test  results.  The  searchers  were  instructed  to  search  each 


of  the  66  questions  in  either  Index  A,  index  3,  or  Index  E  and  to  atteir.pt 
to  locate  all  of  the  relevant  documents  for  each  question.  One  of  the 
searchers  did  not  complete  all  of  the  questions.  One  other  searcher  was 
substantially  better  than  the  ether  searchers  in  his  group  as  can  be 
seen  from  Table  1.  This  searcher’s  results  were  therefore  excluded  from 
the  overall  tabulation. 


Table  1 

Comparison  of  search  results  of  Group  I 
searchers  ana  excluded  searcher 


Recall 
Precision 
Search  time 


Group  I  Excluded  searcher 

mean  mean 

46. 0%  u7.8% 

26.4%  56.0% 

15.22  mil..  9.91  min. 


All  of  the  66  questions  were  searched  in  Index  A,  the  control  index. 
Eleven  out  of  66  questions  were  also  searched  in  Index  B.,  These  11 
questions  represented  11  out  of  48  questions  that  could  have  been  searched 
in  Index  B  since  48  out  of  the  66  questions  were  searched  by  the  two 
chemists  who  added  keywords  to  their  document  titles.  These  11  questions 
were  selected  because  they  had  relevant  documents  that  could  be  retrieved 
only  or  more  easily  with  the  ..eywords  added  by  the  chemists.  The  re¬ 
maining  '  questions  were  also  searched  in  Index  E. 


The  searchers  were  divided  ir.tc  two  groups,  Group  I  ar.^  Group  II. 

A  different  search  pattern  was  assigned  to  each  of  the  two  groups.  Half 
of  the  searchers,  the  Group  I  searchers,  searched  half  (33)  of  the 
questions  in  Index  A.  The  other  half  of  the  searchers,  Group  II,  searched 
the  other  half  of  tine  questions  ir.  Index  A.  Group  I  searched  the  ques¬ 
tions  that  it  cid  net  search  m  Index  A  ir.  either  Index  E  or  Index  E. 

Group  II  searched  the  23  questions  that  it  did  not  search  in  Index  A  in 
Index  5  or  Index  E .  Thus  the  sare  questions  that  were  searched  by  Group  I 
in  Index  A  were  searched  by  Group  II  ir.  Index  B  or  Index  E  and  vice  versa. 
Each  croup  searched  half  of  the  Index  B  questions  in  Index  B  and  half  of 
the  Index  E  questions  ir.  Index  E.  Each  searcher  therefore  searched  33 
questions  ir.  Index  A,  five  cr  six  questions  in  Index  B  and  27  or  28 
questions  in  Index  E.  The  sear ..h  pattern  for  Group  I  and  Group  II  is 
given  ir.  Table  2.  This  search  pattern  was  intended  to  yield  seven  com¬ 
plete  sets  cf  questions  searched  in  Index  A  and  seven  complete  questions 
searched  in  Indexes  P  and  E.  Less  than  seven,  complete  sets  were  actually 
obtained  since  not  all  anticipated  search  results  were  obtained,  for 
reasons  that  have  already  been  indicated. 

c.  Measures  of  search  results 

The  commonly  used  recall  (number  of  relevant  documents  retrieved 
over  the  number  of  relevant  documents  in  collection)  and  precision  (num¬ 
ber'  of  relevant  documents  retrieved  over  total  number  of  documents 
retrieved)  index  measures  were  used  in  the  test.  Relevance  judgment  by 
the  original  searchers ■  the  chemists  who  conducted  the  searches  in  their 
own  document  collections  and  for  thoir  own  use,  were  used  in  the  calcula¬ 
tion  of  recall  and  precision.  Each  of  the  searchers  in  the  test  was  also 


asked  to  k.ep  a  record  of  search  tine  for  each  question. 


Table  2 

Search  Pattern  for  Group  1  and  Group  II 


Indexes  searched 


— — - 1 

1 

r  - 

A 

B 

- 1 

A 

E 

Group  I 

questions 

questions 

questions 

questions 

searchers 

1-6 

7-11 

12-17, 

18-23, 

24-42 

43-66 

Group  II 

questions 

questions 

questions 

questions 

searchers 

7-11 

1-6 

18-23, 

12-17, 

43-66 

24-42 

The  commonly  used  recall  and  precision  measures  are  less  than 
satisfactory  for  the  comparison  of  indexes .  For  example ,  Fairthorne 
points  out  that  neither  the  recall  nor  the  precision  ratio  tell  how  many 
relevant  documents  were  retrieved.0  Yet  a  20%  precision  may  be  perfectly 
acceptable  when  two  relevant  documents  are  retrieved  and  unacceptable 
when  20  relevant  documents  are  retrieved.  The  comparison  of  indexes  by 
means  of  curves  of  recall  versus  precision  is  not  as  useful  as  might  be 
desired.  Is  an  index  with  50%  recall  and  80%  precision  better  than  an 
index  with  40%  recall  and  90%  precision?  We  do  not  really  know.  Search 
results  were  therefore  also  characterized  in  terms  of  three  single  figure 


measurements  to  determine  whether  or  nou  they  are  more  satisfactory  than 


the  recall  ar.d  precision  measures. 

The  first  single  figure  measure  used  is  called  E  (efficiency)  by 
Swets,  who  bases  his  measure  on  statistical-decision  theory.  The  con¬ 
ditional  probability  of  a  hit  is  plcctea  against  the  conditional 
probability  of  a  false  drop  and  measured  by  means  of  an  operating 
characteristic,  the  normal  deviate  to  obtain  E. 

The  second  single  figure  measure  is  called  ERF  (expected  search 
length  reduction  factor) .  The  measure,  which  was  developed  by 
Cooper,  is  based  on  the  calculation  of  the  expected  number  of  non- 
relevant  documents  which  would  have  to  be  looked  at  before  the 
relevant  documents  could  be  found.  Retrieval  performance  of  an  index 
is  evaluated  against  random  searching  through  a  document  file.-'  For 
both;  E  end  ERF  meat.  :res,  the  higher  the  measure  the  better  the  index. 

The  actual  time  required  for  conducting  searches  in  an  index 
appears  to  be  an  important  index  variable  and  one  that  has  not  as  yet 
been  studied  to  any  extent.  If,  for  example,  one  index  is  better  in 
terms  of  precision  but  worse  in  terms  of  search  time  than  another 
index,  the  advantage  of  better  precision  may  be  cancelled  by  the  dis¬ 
advantage  of  slower  search  time.  A  single  figure  index  measure  and 
one  that  is  h *sed  on  recall,  precision  as  well  as  actual  search  time 
was  therefore  sought.  In  the  course  of  looking  for  such  an  index 
measure,  a  literature  search  was  made  and  a  potentially  relevant  docu¬ 
ment  was  located.  The  document  is  by  Lazorick  and  Minder  and  it  is 
on  the  least  cost  searching  sequence  for  "erifying  orders  of  books  in 
a  library6.  The  p-.  i  son  verifying  orders  has  the  choice  of  several  tools  or 
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bibliographies  to  do  this  task.  Ke  decides  on  one  tool  first,  and  if 
this  tool  does  not  have  the  bibliographic  citation  of  the  requested  item, 
he  must  go  to  a  second,  third,  and  subsequent  tool  until  he  finds  the 
wanted  iuj.ormation.  Lazorick  and  Minder  suggest  that  the  choice  of  the 
first  tool  and  of  the  i.  jbsequent  tools  (the  search  sequence)  should  be 
based  on  the  time  that  it  takes  to  search  a  tool  as  well  as  on  the  likeli¬ 
hood  of  locatinn  the  sought  bibliographic  citation  in  that  tool.  This 
is  expressed  as  the  following  ratio: 


Searching  sequence  ratio 


Time  for  searching  a  tool 
Probability  of  success 


The  tool  with  the  lowest  searching  sequence  ratio  is  the  one  to  search 
first,  the  tool  with  the  next  lowest  ratio  is  to  be  searched  second,  etc. 

There  seem  to  be  some  similarities  between  the  best  bibliography  for 
locating  and  verifying  a  bibliographic  citation  and  the  best  index  for 
locating  a  set  of  documents.  In  both  cases  we  are  interested  in  a  high 
probability  of  success  (high  recall)  to  be  obtained  in  as  s'^ort  a  search 
time  as  possible.  There  are  also  differences  between  these  two  types:  of 
searches,  the  most  important  one  being  hat  in  the  case  of  the  biblio¬ 
graphy  one  and  only  one  document  is  sought  and  t  is  either  found  or  not 
found.  In  index  searches,  one  or  more  documents  are  sought  and  m 
addition  to  retrieving  the  wanted  documents,  r 'n-wanted  documents  (false 
drops)  may  be  retrieved  also.  The  set  h  time  and  the  probability  of 
success  (recall)  in  the  searching  sequence  ratio  can  therefore  be  used 
for  characterizing  part  of  the  search  results  but  an  additional  factor 
must  be  added  for  the  retrieval  of  non-wanted  documents,  the  false  drops. 
We  can  do  this  by  assigning  a  demerit  : or  each  false  drop  if  we  argue 


12 


that  false  drops  add  to  the  searching  time  of  an  index.  The  modified 
searching  sequence  ratio,  called  time-recall  factor  (TRF)  is  given  belowj 

Time  for  searching  a  tool  +  (number  of  false 

.  drops  x  time  to  remove  false  drop) 

Time  recall  factor  =  — — r— - - - , - — - r 

Probability  of  success  (recall  probability) 

The  time  for  removing  a  false  drop  is  a  variable  since  it  will  differ  in 
different  indexing  systems.  The  time  is  dependent,  among  other  things, 
or.  the  time  it  takes  to  locate  the  document  and  the  time  it  takes  to 
determine  that  it  is  a  fal^e  drop.  We  have  used  a  figure  of  two  minutes 
in  our  tests  of  the  indexes  since  it  is  our  estimate  that  it  takes  about 
a  minute  to  locate  and  refile  the  document  and  about  a  minute  to  determine 
that  it  is  a  false  drop.  It  should  be  pointed  out  that  the  lower  the  TRF, 
the  better  the  index. 

III.  Search  results 

a.  Group  I  and  Group  II;  Index  A  and  Index  B;  Index  A  and  Index  E 

In  this  part  of  the  report,  comparative  search  results  are  given 
for  the  two  groups  of  searchers,  Group  I  and  Group  TI ,  and  the  index 
variables  tested  -  the  added  keywords  (Index  B)  and  the  single  access 
point  per  document  alphabetic  subject  index  {Index  E) ,  both  tested  against 
Index  A.  The  results  are  characterized  an  terms  of  recall,  precision, 
search  Time,  TRF,  ERF,  an!  E  in  Fiqs.  3-20.  In  each  figure  the  mean 
search  results  per  question  for  the  groups  of  searchers  as  well  as  the 
overall  mean  for  the  aroupsof  searchers  are  oiven.  The  data  in  Figs.  3- 
20  are  summarized  in  Table  3.  The  search  results  are  analyzed  for 
statistically  siunif leant  differences  with  the  aid  of  two  tests;  the 
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analysis  of  variance  and  the  signed  pair  testa.  The  signed  pair  test 
was  also  used  by  Salton  in  his  analysis  of  search  results  obtained  in 
his  machine  prepared  and  searched  indexes. ^  The  analysis  of  variance 
tests  were  run  with  the  aid  of  a  computer  program.8  Results  of  these 
tests  of  statistically  significant  difference  are  given  in  Table  4. 
b.  Index  C  and  Index  D 

Neither  Index  C,  the  index  with  only  16  delete  words,  nor 
Index  D,  the  index  with  elements  of  vocabulary  control,  were  actually 
prepared  in  their  final  form.  These  indexes  were  therefore  not  searched 
by  the  graduate  chemistry  students.  The  anticipated  size  of  Index  C  and 
computer  program  problems  in  the  preparation  of  Index  D  discouraged  us 
from  preparing  these  two  indexes.  Instead,  simulated  rather  than  actual 
searches  were  made  with  these  indexes.’ 

For  Index  c,  titles  of  relevant  documents  were  checked  against  the 
corresponding  questions  to  determine  whether  any  of  the  words  deleted  in 
Index  A  but  not  deleted  in  Index  C  would  have  retrieved  the  relevant  docu¬ 
ments.  Five  such  delete  words  from  Index  A  were  located.  Table  5  lists 
these  five  delete  words  along  with  the  number  of  relevant  documents  that 
might  have  been  retrieved  with  each  word.  The  total  number  of  relevant 
documents  for  each  of  the  questions  for  which  the  delete  word  might 
have  been  useful  is  given  in  parentheses. 

For  Index  D,  titles  of  relevant  documents  were  checked  against  the 
corresponding  questions  to  determine  how  many  additional  relevant  docu¬ 
ments  might  have  been  retrieved  with  the  aid  of  the  cross-references  that 
were  to  be  provided  in  this  index.  The  12  cross-references  in  Index  D 
that  might  have  been  of  assistance  in  selecting  additional  relevant 
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Results  of  Tests  of  Statistical  Difference 
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Symbol  in  parentheses  indicates  which  index  or  group  is  better. 


documents  are  listed  in  Table  b.  This  table  lists  the  cross-references 


and  the  number  of  additional  relevant  documents  that  might  have  been 


Table  5 


Index  A  -  delete  words  that  might  have  been  used 
to  retrieve  relevant  documents 

Delete  word  Number  of  documents  that  might  have  been  retrieved 


theory 

forming 

studies 

element 


elements 


1  (of  16  relevant) 
i  (of  2  relevant) 

1  (of  12  relevant) 

2  (of  11  relevant) 
1  (of  20  relevant) 


retrieved  with  the  aid  of  these  cross-references  for  12  out  of  the  66 
questions.  It  should,  however,  be  pointed  out  that  the  searchers  did 
retrieve  some  of  these  documents  with  Index  A,  an  index  without  cross- 
references.  From  this  we  can  conclude  that  in  at  least,  some  of  the  cases, 
the  searchers  directed  themselves  to  tha  rppropria  *ords  without  the 
benefit  of  cross-references. 

TV.  Discussion  of  results 

a.  Index  A  and  Index  B,  and  Index  A  and  „ndex  E 

No  statistically  significant  difference  was  found  with  either 
the  analysis  of  variance  or  signed  pair  rest  for  the  ix  sets  of  search 
results  obtained  by  the  Group  I  and  Group  II  searchers  We  can  therefore 
conclude  that  any  difference  between  indexes  is  not  attributable  to  the 
difference  between  the  groups  of  searchers. 

No  significant  difference  was  found  with  the  analysis  of  variance 


test  for  the  five  sets  of  search  results  for  Index  .  and  Index  B  that 
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Table  6 

Cross-references  for  Index  D  that  might  have  >->een  used  to 
reprieve  relevant  documents 


Keyword 

Crosi'S-reference 

Additional 

relevant  documents 

lanthanides 

rare -earths 

2 

lanthanide-sulfate 

ra_  a-earth-sulfates 

1 

heat-data 

thennodynaro  l  c  s 

1 

1  lanthanides 

rare-earths 

i 

lanthanide 

rare-earth 

2 

lanthanides 

rare-earthe 

7 

spectroscopy 

spectra 

2 

r are-earths 

lanthanides 

i 

lanthanide 

rare-earth 

i 

lanthanides 

lanthanons 

i 

ion-ex  change 

cation-exchange 

5 

ion-exchange 

anion- -exchange 

.4. 

resm 

polymer 

2 

structure -of -water 

water- structure 

5 

structure -of: -water 

water-structure 

t. 

were  compared .  No 

analysis  of  variance  woa 

run  on  the 

TRF  results  since 

only  seven  out  of  11  pairs  of  search  results  could  be  compared.  Reasons 
for  tills  will  be  explained  in  the  discussion  of  t ha  TRF  measure.  There 


is  a  statistically  significant  difference  at  the  0.05  level  in  the  signed 
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ran:  .est  for  both  recai 1  a--  3  precision.  Eight  and  nine  out  of  11  pairs, 
respectively  search  results  for  recall  arid  pre.  sion  show  Index  E  to 
be  better  than  Index  V  There  is  no  statistically  significant  difference 
at  the  0.01  level  for  either  recall  or  precision  between  Index  A  and 

Index  B, 

In  the  comparison  of  Indexes  A  and  E,  only  the  search  time  for  Index 
£  is  significantly  better  at  the  0.05  level  with  the  analysis  of  variance 
test,  this  difference  does  not  hold  at  the  C.01  level.  All  the  other 
Starch  results  for  Index  A  and  E  for  both  analysis  of  variance  and  signed 
pair  rests  show  no  statistically  significant  difference  between  these  two 
indexes . 

Even  though  the  statistical  tests  search  results  axe  in  agreement 
in  14  out  of  17  cases,  some  caution  in  ^rawing  conclusion®  from  these 
results  is  in  order.  The  analysis  of  variance  test  assumes  normal  distri- 
bu’ion  of  search  results,  something  that  not.  obtained  as  can  be  seen 
from  Figs.  3-20.  The  non-normal  distribution  of  search  results  is  par- 
ticularl’  pparent  in  the  recai 1  results.  This  is  due  at  least  .n  part 
to  the  large  number  of  searches  that  had  only  one  relevant  docvjnent. 

When  this  was  the  case,  either  100%  recall  was  achieved  when  the  relevant 
document  was  retrieved  or  0%  recall  was  achi»-  ed  when  the  relevant  docu¬ 
ment  was  not  retrieved 

The  signed,  pair  test  is  baaed  on  any  difference  between  pairs  of 
search  results  whether  this  difference  is  a  fraction  of  a  percentage 
point  or  20  or  more  percentage  points.  There  is  a  question  whether  a 
fraction  of  a  percentage  point  difference  is  meaningful.  Yet  we  cannot 
set  the  minimum  meaningful  difference  et  a  specified  quantity,  partly 
becaus_.  •  e  do  not  know  at  this  time  what  this  quanti  ty  should  be  and 


I 
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partly  because  if  we  did  do  this  we  would  run  the  risk  of  introducing  a 
bias  m  the  test. 

We  anticipated  Index  A  to  per  for  i  better  than  Index  E  since  Index  A 
had  mere  access  points  per  document  than  Index  E.  The  following  reasons 
might  explain  why  Index  A  did  not  perform  significantly  better  than  Index  E. 

*  The  additional  information  per  alphabetic  subject  index  entry'  as 
compared  with  the  KWOC  index  entry,  the  subject  heading,  may  have  offset 
the  disadvantage  of  only  a  single  access  point  per  document.  This  might 
have  been  the  case  for  noth  the  selection  of  relevant  document*,  and  the 
rejection  of  non-relevant  documents . 

*  There  are  relatively  few  subject  headings  in  the  index,  only  194, 
and  relatively  few  documents  under  each  subject  heading  as  can  be  seen 
froit  Fig.  21.  This  may  account  for  Index  F's  advantage  in  search  time 
since  less  time  is  required  to  locate  the  access  point  in  this  index  than 
with  Index  A.  Another  .reason  for  the  better  search  time  tor  Index  E  is 
that  fever  access  points  were  searched  per  question  in  Inuex  E  than  in 
Index  A. 

Index  B  had  statistically  significant  Letter  recall  and  precision 
than  'ndex  A  with  the  signed  pair  test  though  not  with  the  analysis  of 
variant-  test.  Index  B  was  expected  tr  show  better  recall  and  precision 
since  it  included  the  added  keywords  chosen  by  the  two  chemists  for  the 
indexes  to  their  documents .  The  better  recall  car.  be  explained  by  the 
additional  access  points  per  document  (a  factor  that  was,  however,  of 
questionable  value  in  the  Index  A  versus  Index  E  comparison) .  The  added 
keywords  ♦. agged  along  with  every  index  entry,  we  ther  or  not  they  were 
the  access  points.  This  added  information  per  entry  mav  have  been  of 
use  in  eliminating  iron-relevant  documents  and  may  therefore  explain  the 
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better  precision  of  Index  B  than  Index  A.  No  better  (faster)  search  time 
v;as  either  expected  or  obtained  with  Index  B  since  the  added  keywords 
would  have  a  tendency  to  lengthen  the  search  tine  if  they  affect  search 
time  at  all.  Forty-eight  questions  out  of  the  66  questions  came  from  the 
two  chemists  who  added  keywords  to  their  indexes.  Forty-eight  questions 
could,  therefore,  have  been  searched  in  Index  B  instead  of  the  11 
questions  that  were  actually  searched  in  it.  Only  11  questions  were 
searched  in  Index  B  since  only  these  qu  -ticr.s  were  searched  that  were 
considered  tc  b°  easier  to  search  wich  this  index.  These  are  the  ques¬ 
tions  for  which  useful  access  points  could  be  supplied  better  or  only 
by  means  of  the  added  keywords.  Had  all  *.8  questions  been  searched  in 
Index  E,  chances  are  that  the  difference  in  performance  between  the  index 
without  added  keywords  and  the  index  with  added  keywords  would  have  been 
even  smaller. 

b.  single  figure  index  measures 

Since  none  of  the  comparative  search  results  with  measures  E, 
ERF,  and  TRF  on  the  groups  of  searchers  end  the  three  indexes  showed  any 
statistically  significant  difference,  no  comments  car.  tc  made  or.  tire 
discriminating  power  or  the  three  index  measures.  In  looking  at  Table  3 
we  can  see  that  the  E  and  ERF  measures  are  consistently  higher  (better) 
lor  the  two  groups  of  searchers  and  tire  two  pairs  of  indexes  tested. 

The  T  rK  measure  was  also  consistently  lower  (better)  for  the  two  croups 
of  searchers  and  for  Index  A  versus  Index  B.  The  TRF  was  better  tor 
Index  E  rather  than  Index  A  as  was  the  case  for  both  the  E  and  ERF 
measures.  This  result  is  probably  attributable  to  the  better  search 
time  obtained  with  Index  E. 
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There  were  some  problems  in  using  the  single  figure  index  measures 
and  these  problems  will  now  be  discussed.  An  adjusted  value  of  99.99% 
recall  instead  of  100%  had  to  be  used  for  calculating  E  values  for  search 
results  with  100%  recall,  since  the  percentage  of  false  drops  was  quite 
low  in  most  cases,  a  graph  paper  other  than  the  one  originally  suggested 
by  Swets  had  to  be  used.  Dr.  Swets  kindly  supplied  a  different  graph 
paper,  K-E  47-8062,  that  could  be  used.  It  was  not  possible  to  determine 
the  slope  for  each  search  both  because  a  large  number  of  questions  yielded 
only  one  relevant  document  and  because  of  the  small  number  of  searches 
per  question.  Instead,  average  figures  were  used  for  number  of  relevant 
documents  retrieved,  number  of  relevant  documents  not  retrieved,  number 
of  non-relevant  documents  retrieved,  and  number  of  non-relevant  documents 
not  retrieved.  The  measure  E  was  derived  by  the  use  of  the  operating 

i- 

q 

characteristic  curve  published  in  the  Swets  paper. 

For  questions  that  yielded  only  one  relevant  document  and  one  that 
was  not  retrieved,  a  negative  value  was  obtained  with  the  ERF  measure. 

This  means  that  for  this  particular  type  of  search,  the  search  time  with 
an  index  takes  longer  than  purely  random  retrieval.  This  may  or  may  not 
be  so. 

The  TRF  measure  could  not  be  used  with  0%  recall  search  results. 

Such  search  results  yielded  an  unrealistic  TRF  value  of  infinity.  Search 
results  with  0%  recall  were  therefore  emitted  from  the  statistical  analysis, 
hence  the  smaller  number  of  paired  search  results  that  were  compared  for 
the  TRF  measure  than  for  other  search  measures.  To  get  around  this  problem, 
the  recall  at  both  extreme,  0%  and  100%,  might  be  adjusted  upward  and 
downward,  respectively.  The  two  minute  time  for  identifying  and  rejecting 
a  false  drop  may  or  may  not  be  realistic.  This  figure  can  be  adjusted  to 
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represent  the  tine  tor  any  particular  indexing  system, 
c.  Index  C  and  Index  D 

The  KWGC  index  that  was  to  have  only  16  delete  words  instead 
of  the  u2n  delete  words  wo  ud  have  been  a  very  large  index  indeed, 
several  tines  larger  than  the  index  with  628  delete  words.  In  this 
instance  at  least,  cr.e  does  not  get  much  from,  the  extra  bulk.  Only 
six  relevant  documents  that  were  net  retrieved  with  Index  A  night  have 
been  retrieved  with  the  minimum  delete  word  index.  Furthermore,  it  is 
questionable  whether  a  searcher  would  look  under  two  of  the  words  that 
night  have  retrieved  the  additional  relevant  documents,  'the  words 
"fermina"  and  "studies". 

An  examination  of  the  search  results  obtained  with  Index  A,  the 
index  without  elements  of  vocabulary  control,  indicated  that  the  searchers 
of  this  index  thought  of  a.-.c  used  seme  c:  the  cross-references  that  were 
to  be  provided  :n  Index  2.  "he  extent  c:  this  generation  of  cross-refer¬ 
ences  by  the  searchers  will  be  det»rr  ir.ed  from  a  further  analysis,  of  •■he 
search  results. 


d.  Some  general  conclusions  and  suggestions  icr  future  work 

To  anyone  acquainted  with  index  testing,  our  general  conclusion 
will  not  come  as  a  surprise:  there  are  problems  in  index  testing  with 
which  we  have  net  as  yet  learned  to  ccpe ,  Some  of  these  croblems  alone  [th 
comments  on  what,  if  arvthmg,  was  done  about  them  in  our  tests  of  the 
indexes,  will  now  be  summarized. 

1.  The  subjectivity  and  inconsistency  -wf  relevance  judgments 
have  been  problems  m  previr  £  tests  c;'  indexes,  ar.d  these  problems  have 
rot  been  resolved  in  our  tests.  Two  steps  have  been,  taken  to  reduce  the 
problem,  of  relevance  judgment.  Relevance  judgments  by  the  original 
searchers,  the  three  chemists  who  searched  their  cvr.  document  collections, 
were  used  for  the  characterization  of  relevant  documents .  S-  archers  with 
similar  educational  background,  graduate  chemistry  students,  were  employed 
to  conduct  the  test  searches.  Msc ,  search  results  by  groups  of  searchers 
rather  than  by  individual  sear  hers  were  used  in  the  .*r.a lysis  cf  the  test 
results . 

1.  There  is  a  subtle  but  important  distinction  '  e tween  a 
statistically  significant  difference  ar.d  a  reanmefui  difference  ir-  the 
performance  of  two  indexing  systems,  ffatistics  enable  us  to  determine 


whether  a  difference  is  "ere  than  a  matter  or  chance.  but  statistical ! v 
significant  differences  ray  or  may  not  be  meaningful  differences  from 
the  point  of  v . -v  vf  the  overall  per formant «  of  the  indexing  sv s t  ems . 
This  truest  ion  will  remain  unresolved  until  we  test  the  overall  indexing 


s  vs  ter  rather  than,  the 


the  svster  ar.c  until  we  can  s:  «•;  ify 


what  1b  a  meaningful  di  r  t e: or  •:  e  between,  index ino  svs terns  . 


Searrn  res-; Its  ‘'or  •:  cstifns  with,  cr.l’ 


re  . evar.: 
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ment  give  extreme  recall  values  of  0%  when  the  document  is  ..^t  reprieved 
and  10C%  whan  the  document  is  retrieved.  These  single  relevant  document 
search  results  may  carry  an  unduly  large  weight  in  averaging  search  re¬ 
sults.  Search  results  for  searches  with  one  relevant  document  also- 
caused  difficulties  in  calculating  each  of  the  three  single  figure  measures 
of  index  performance.  The  number  of  relevant  documents  per  question 
should  perhaps  be  one  of  the  index  variables  to  be  tested.  We  plan  to 
analyte  separately  the  search  results  for  the  *1  questions  that  had  only 
one  relevant  document. 

4.  All  other  things  being  equal,  the  cost  of  preparation  of  an 
index  with  elements  o'  vocabulary  control  is  greater  than  the  cost  of 
preparation  of  an  index  without  elements  cf  vocabulary  control.  Again, 
with  all  other  things  being  equal,,  an  index  with  elements  of  vocf  clary 
control  is  likely  to  give  highe  ...call  than  an  index  without  elements 
of  vocabulary  control.  However,  some  evidence  has  been  collected  in  the 
course  of  this  study  that  the  searchers  will  think  of  and  supply  their 
own  cross-references  in  an  index  without  elements  of  vocabulary  control. 
Further  work  is  therefore  suggested  to  determine  both  the  cost  and  the 
benefit  of  elements  of  vocabulary  control  in  an  indexing  system, 

Tnere  is  an  indication  that  for  a  collection  of  about  3200  docu¬ 
ments  a  single  access  point  per  document  alphabetic  subject  index  is  as 
good  as  a  multiple  access  point  per  document  keyword  from  title  index. 

We  would  like  to  see  these  findings  tested  on  indexes  that  are  larger  by 
an  order  of  magnitude.  This  could  be  done  with  the  66  questions  that 
were  used  in  this  study  by  modifying  an  existing  index,  Chemical  Abstracts 
Service'  Chemical  Titles.  The  magnetic  tapes  for  several  issues  of  this 
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keyword  fror.  title-  index  would  he  merged  and  a  merged  index  to  about 
30 , 0C0  docur.er.ts  would  be  printed.  The  resulting  keyword  fror  title 
index  would  th.en  he  compared  with  a  single  access  point  per  document 
index  to  the  same  group  of  documents.  The  sir  ale  access  point  per 
document  index  would  be  the  listing  of  the  document  citations  under 
the  sections  headings  used  in  Chemical  Ahs^.acts  for  grouping  the 
abstracts  of  the  documents . 
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QUESTIONS  FOR  KWOC  INDEX  TEST 
A  chromatographic  method  for  purifying  lipids. 

Want  an  experimental  method  for  preparing  methyl  esters  of  fatty 
acids . 

Experimental  details  concerning  an  extract  of  penicillium  which 
converts  rnethyisalicyiic  acid  to  patulin. 

Identification  of  methyl  derivative  of  triacetic  aciu  xactcne  in 
penicillium  stipitatum. 

Heterocaryon  formation  and  genetic  analysis. 

What  is  the  content  of  fatty  acids  in  brain  .Lipids? 

6-methyl  salicylic  acid  production  in  penicillium. 

Calculate  quantity  of  rnethyisalicyiic  acid  produced  xn  a  cell-free 
system . 

Concentration  of  triacetic  acid  la "tone  used  to  stimulate  aromatic 
biosynthesis . 

Method  of  separation  of  various  unsaturated  fatty  acids. 

Growth  properties  of  penicillium  patulin  in  relation  to  meth> 1- 
salicylic  acid  biosynthesis. 

Information  on  the  sulphate  complexes  of  lanthanide  elements. 
Information  on  the  stability  constant*,  of  lanthanide  sulfate  com- 
plexing. 

Heat  data  for  xare-earth  acetates. 

Method  of  preparation  of  anhydrous  salts  in  connection  with  spectro- 
pho tome trie  studies  of  lanthanides. 

Information  on  the  theory  for  intensity  of  lanthanide  spectral  lines. 
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IV.  Spectroscopy  of  lanthanides. 

18.  Stability  data  on  chelate  complexes  cf  rare-earths . 

19.  How  are  the  hydrates  involved  in  the  method  of  preparation  of 
volatile  complexes  of  lanthanide  elements? 

20.  Information  on  stability  cf  volatile  complexes  of  the  lanthanide 
elements . 

21.  Information  on  the  thermodynamics  of  water  absorption  in  order  to 
start  writing  a  paper  on  nuclear  magnetic  resonance  studies  of 
watr*1'  in  ion  exchange  resins. 

22.  What  is  the  structure  cf  water? 

23.  Does  the  resin  matrix  ..ruse  the  'tructure  of  water  to  increase? 

24.  Method  for  preparation  of  diazoethane. 

25.  Chemical  synthesis  of  phospholipid  derivatives. 

26.  Methods  of  preparing  carbonyls  in  aqueous  solutions  in  order  to  get 
references . 

27.  The  use  of  borontr i f luoromethonyl  solutions  for  tre.nsesterifying 
lipids 

28.  Use  of  fructose  d.i -phosphate  in  stabilizing  the  fatty  acid  synthetase. 

29.  What  are  the  experimental  details  in  the  use  of  dithiothreitol  in 
the  protect. on  of  enzymes? 

30.  References  to  article  that  uses  dithiothreitol  in  stabilizing  the 
pigeon  liver  fatty  acid  synthetase. 

31.  Conditions  for  forming  acetyl  derivatives  of  acetyl  earlier  protein. 

32.  Reference  to  a  paper  which  would  have  a  synthesis  of  acetoacetyl 
eoenzyme-A . 

33.  Metal-metal  bend  energy  of  dimanganese-carbonyl . 

34.  Htabi  1  i  *  y  ;  roper  ties  of  *  n  acetic  acid  and  its  lactones. 
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35.  .Method  of  preparation  of  4 -hydroxy -6 -methyl  pyridone  and  properties. 

36.  Degradation  scheme  for  the  compound  patulin  in  order  to  test  the 
feasibility  of  an  isotope  experiment. 

37.  Occurrence  of  hydrocarbons  in  Sarcine  lutea. 

38.  Absorption  spectra  of  dithiothreitol . 

39.  Method  for  making  trimethylsilyl  derivatives  of  sugars. 

40.  Quantity  of  patulin  produced  in  cell  free  system. 

41.  Analysis  of  fatty  acids  of  marine  organisms. 

42.  Want  information  on  the  substitution  of  phosphorus-trifluoride  into 
cobalt- tricarbony 1-nitrosyl . 

43.  Quantity  of  alternarioi  produced  in  a  cell-free  system. 

44.  Medium  for  growing  Alterna. ia  tenuis. 

45.  Experimental  procedure  used  in  purifying  an  enzyme  from  alternarioi. 

46.  Wanted  paper  chromatography  system  to  analyze  triacetic  acid. 

47.  Retention  time  of  hydroxy  fatty  acids  on  gas-liquid  chroma tograpny . 

48.  Concentration  of  methylsalicylic  acid  which  shows  inhibitory  effects 
on  methylsalicylic  acid  synthesis. 

49.  Method  of  isolation  of  penicillium  patulin  metabolites. 

50.  Conditions  used  in  studying  inhibition  of  enzymes  by  sulfyhydryl 
reagents . 

51.  The  tate  of  water  in  ion  exchange  resins. 

52.  The  cumulation  of  side-products  n  fatty  acid  synthesis. 

53.  Identification  of  a  side-product  in  fatty  acid  synthesis. 

54.  Information  on  the  fission  cross-section  as  a  function  of  mass-number. 

55.  Conditions  for  C0o  malonyl  coenzyme-A  exchange  reaction. 

56.  Experimental  procedure  for  enzymatic  assay- 


S'7.  Empirical  Ep  functions  in  t  onuection  with  charge  distribution 
studies  on  fission. 

58.  Nuclear  magnetic  resonance  studies  on  ores,  minerals  and  clays. 

59.  Information  for  preparation  of  a  computer  program  on  stability 
constants . 

60.  Need  inform,  -ioj  for  a  talk  on  new  element  research. 

61.  Experimental  procedures  for  the  '  ihn-Foth  oxidation  wanted. 

(  ...  Conditions  for  obtaining  the  crude  fatty  acid  synthesis  from  E. 

Ceil  . 

6".  Kr '-tv  ter  .nal  ethyl  group  determination  by  chromic  acid 

.  -.idatiun. 

64.  i.ffr  1  oi  .  tfiior.  ne  or.  synthesis  of  orseliiruc  acid. 

65.  Need  method  for  synthesis  of  crsellir.i-.  acid. 

66.  s  tilth,  at  ion  of  metaoolitr  production  m  penioil ;  ium  urticae  barnier 
by  triacetic  acid  lactone. 
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KWOC  INDEX  TEST  INSTRUCTION: 

A.  Indexes  A  and  B 

1.  Introduction 

The  objective  of  this  test  is  t r  determine  how  useful  a  KWOC' 
(Keyword-Out-Of -Context)  index  is  in  the  icc  ting  of  al.  documents  ; n  a 
collection  which  are  relevant  to  a  fart'cul  r  question  through  the  us  a 
of  ’-eywords  in  the  document  .itles.  A  KV.r  index  is  .  computer  print-c 
in  which  significant  words  from  ti  1«  ,  j-es  oi  author.  ,  and  in  some 
cases  additional  keywords ,  are  arrar. _  n  1;  hubetically  .-t  the  it  t-hand 
margin.  In  the  index  you  will  be  us  ing,  t  a  complete  citation  ar.d  docu¬ 
ment  serial  numbers  are  listed  with  the  alphabetized  titl-  words .  At 
the  front  of  the  index  is  a  listing  of  "stopwcrds"  wh. i ch.  are  words  not 
considered  significant  and  not  indexed  as  keywords. 

The  keywords  in  this  index  are  nor.  always  in  a  str  ict  alphabetic 
order  because  of  a  machine  idiosyncracy  ir.  the  processing,  t  .  ■  u.s  ance 
tiie  hyphen  acts  as  if  it  were  a  letter  of  the  alphabet  which,  won  id  f  '  - 
low  the  letter  A.  Therefore ,  vapor -pressure  appear;;  alter  vapors,  ana 
F -orbitals  appear  after  all  the  o tiler  words  beginning  with  the  letter  F. 

2 .  .General  Instructions^ 

You  will  be  given  a  list  f  search  questions  11.  h  of  these 
questions  will  have  one  or  more  documents  that  were  cons  ’ered  pertinent 
to  the  question  by  the  researcher  who  original ly  inked  the  ’uestion. 

You  are  asked  to  select  keywords  under  which  to  look  for  pote.  tially 
pertinent  titles  and  then  decide  whether  the  title  is  pci  ti  .tent .  Your 
mist  be  based  cn  the  words  in  the  title .  You  are 


decision 
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to  locate  all  titles  in  the  index  that  are  pertinent  to  the  question. 

•Ph'i  procedure  that  you  are  to  follow  is  illustrated  by  the  sample 
question  rn  the  form  attached  to  these  directions. 

a.  Record  the  time  at  the  start  of  the  search. 

b.  Read  the  sample  question. 

c.  "Fluorescence "  is  chosen  aB  a  keyword. 

d.  Write  this  keyword  in  the  space  provided  in  the  section 
titled  "Keyword  Selected." 

e.  Scan  the  alphabetically  arranged  keywords  in  the  KWOC 
ind^y.  to  see  if  "fluorescence"  appears  as  a  keyword.  You  will  see  that 
there  are  11  titles  which  contain  the  word  "fluorescence." 

f.  Read  each  title  and  determine  if  it  might  answer  the  ques- 

tion. 


g.  Several  titles  under  this  keyword  are  considered  to  be 
pertinent.  Write  the  serial  number  of  the  titles  you  consider  to  be 
portinent  in  the  space  provided.  If  there  had  not  been  any  titles  con¬ 
sidered  pertinent,  you  would  have  written  "none"  in  the  serial  number 
space. 

h.  Choose  the  next  keyword  that  you  will  use  to  search.  Any 
keyword  may  be  used  in  the  search.  It  does  not  need  to  be  in  the  ques¬ 
tion,  but  can  be  a  synonym  or  a  more  generic  or  specific  form  of  a  word. 
It  may  be  necessary  to  look  under  various  forms  of  the  same  word  when 
locating  information.  For  example,  "fluorescence-spectra,”  a  specifi¬ 
cally  related  form  of  "fluorescence,"  is  chosen  as  a  keyword.  Write 
this  in  the  section  provided  for  keywords.  Turn  in  the  index  to  the 
appropriate  keyword  section.  A  total  of  six  titles  which  contain  the 
word  "fluorescence-spectra"  are  listed.  Read  each  title.  The  title* 
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with  the  serial  number  FV-028  is  selected  as  being  pertinent.  Further 
titles  which  are  pertinent  to  this  question  can  be  found  under  the  key¬ 
words  "fluorescence-spectrum , H  "Europium-hexaf luoroacty lacetonate , " 
"fluorescent,"  and  "Europium-chelate."  Many  of  the  same  titles  found 
under  "fluorescence,"  etc.,  are  also  found  under  the  keyword  “Europium." 

Do  NOT  record  a  document  serial  number  more  than  once  for  any  one  question. 

i.  Continue  your  search  for  each  question  until  you  feel  you 
have  selected  ALL  pertinent  documents  to  the  question  or  have  determined 
that  no  pertinent  document  can  be  found . 

j.  Record  the  time  when  you  finish  searching  on  this  question. 

As  has  been  stated  above,  it  may  be  necessary  to  look  under  various 

forms  of  the  same  word  when  locating  information.  For  example,  if  one 
wants  information  on  solubility,  he  can  look  under  "solubility," 
"solubilities,"  "solubilization,"  and  "soluble." 

Following  is  a  test  question  to  be  searched  in  order  for  you  to 
familiarize  yourself  with  the  index.  Please  feel  free  to  ask  any 
question  you  might  have. 

Thank  you  for  your  cooperation  in  this  test. 

3.  Sample  Question  for  KWOC  Index  Test 

Search  in  Index  A: 

Information  on  ion  exchange  in  connection  with  research  on  water. 

B.  Index  E 

This  index  is  a  subject  index,  with  citations  listed  under  a  subject 
heading  assigned  by  the  researcher. 

At  the  front  of  the  index,  you  will  find  a  typewritten  list  of 
subject  headings.  The  computer  print-out  is  a  list  of  subject  heading 
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codes  arranged  in  alphabetical  order.  All  citations  dealing  with  a 
particular  subject  are  listed  after  each  subject  heading  code. 

In  your  search  of  this  index  you  willt 

1.  Record  the  time  starting  search. 

2.  Read  the  question. 

3.  Look  over  the  list  of  subject  headings  to  determine  under  which 
ones  you  will  search. 

4.  Record  in  the  "keyword"  column  the  subject  heading  code  that 
you  will  search  first. 

5.  Turn  to  that  code  in  the  index  and  read  the  citations  listed. 

6.  Record  the  document  serial  numbers  of  citations  that  appear  to 
be  relevant  to  the  question. 

7.  Repeat  steps  4,  5,  and  6  for  any  additional  subject  headings 
that  might  contain  relevant  documents. 

8.  Search  until  you  feel  ALL  relevant  documents  have  been  located. 

9.  Record  time  of  end  of  search. 

If  we  use  the  sample  search  that  was  done  for  the  instructions  for 
Indexes  A  and  B,  "Need  information  on  fluorescence  of  Europium , "  we  can 
illustrate  the  search  procedure.  After  reading  the  question,  read  over 
the  list  of  subject  headings.  Heading  "FV"  (Lanthanides  -  Spectra)  would 
probably  contain  some  relevant  material.  Enter  "FV”  in  the  keyword 
column.  Turn  to  "FV"  in  the  index  and  read  the  citations  of  the  documents 
listed.  In  the  document  serial  number  column,  enter  the  14  serial  numbers 
of  the  relevant  documents.  On  further  reading  of  the  list  of  subject 
headings,  you  decide  that  heading  "BZ"  (Complexes  -  Spectra)  might  also 
contain  some  relevant  material.  Enter  "BZ"  in  the  keyword  column,  turn 
to  "BZ"  in  the  index,  read  the  document  citations  listed,  and  enter  any 
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document  aerial  numbers  you  feel  are  relevant.  In  this  case,  only  one, 
"BZ-019,"  would  be  listed. 

Please  try  this  search  for  the  sample  question  given  in  the  in¬ 
structions  for  Indexes  A  and  B.  If  you  have  any  questions,  please  ask 
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Three  indexes  to  a  collection  of  3204  documents  in  the  field  of  chemistry  were 
test  searched.  The  indexes  are  a  keyword  from  title  index  without  added  key¬ 
words,  a  keyword  from  title  index  with  added  keywords,  and  a  single  access  point 
per  document  alphabetic  subject  index.  The  three  indexes  were  searched  by  14 
graduate  chemistry  students  with  66  questions .  Search  results  are  characterized 
in  terms  of  recall,  precision,  search  time,  and  three  other  single  figure 
measures.  A  measure  of  index  performance  based  on  recall,  precieion  and  search 
time  was  develonod.  There  is  no  statistically  significant  difference  between 
search  results  with  the  multiple  access  points  per  document  keyword  from  title 
index  and  the  single  access  point  per  document  alphabetic  subject  index. 
Statistically  significant  differences  in  search  results  were  found  between  the 
keyword  from  title  index  with  added  keywords  and  the  keyword  from  title  index 
without  added  keywords.  The  effect  of  the  size  of  the  delete  word  list  and  of 
elements  of  vocabulary  control  in  keyword  from  title  indexes  were  also  studied. 
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